Description

Adding a MultiModal Sharded Dataloader and several of the samplers to be used

Additional context

ShardedDataloader is a flexable multimodal dataloader that attempts to solve the use case of large scale llm/vlm training. A single backed instance of S3 dataloader is not sufficient for the following reasons will face the following issues which this dataloader attempts to address:

• PreDownloading Times: Often times other dataloaders implementations either require you to either predownload parts of whole of the datasets. Even datasets implementation like IndexedDataset require you to generate index files either ahead of time or during runtime which is expensive
• Fast Access: MMap datasets are often infeasable when a customer want to combine many seperate datasets. Without Mmap, access times are slow and with MMAP you can hit the open files limit of the system
• Flexibility: Dataloaders sometimes make the trade off and sacrifice performance by pretokenize/preembed multimodal components. This doesn't scale to complex multimodal types like video and often requires individual to reprocess their data every time a data prep step changes

• I have updated the CHANGELOG or README if appropriate

Related items

Testing

By submitting this pull request, I confirm that my contribution is made under the terms of BSD 3-Clause License and I agree to the terms of the LICENSE.